Support CPython 3.11, 3.12, and aarch64 processors #2331

ddelange · 2023-01-20T14:54:17Z

Hoi 👋

linux-aarch64 makes up for almost 10% of all platforms ref giampaolo/psutil#2103

aarch64 has already surpassed windows in terms of downloads for this package. Oracle, Amazon, Google, and Microsoft are all offering aarch64 cloud instances at an undeniable price point compared to amd/intel, so the demand will undoubtedly only grow

this PR is adapted from Add arm64 mac and linux wheels MagicStack/asyncpg#954
uses QEMU emulation for linux arm64 wheels: manylinux takes around 2.5hrs per wheel and alpine arm64 up to 4 hrs 😅
manylinux2014 wheels are built with GCC 10, which I think does not guarantee proper functioning of pybind11 (docs).
- so with this PR, linux wheels are built with GCC 12 (manylinux_2_28).
- pip will only install these wheels on linux operating systems with glibc >= 2.28 (mostly all 2020+ linux distributions like debian 10 buster, ubuntu 20.04 focal, almalinux/rhel 8, ...).

the wheels from this PR can be installed with:

# comma separated list for --find-links
export PIP_FIND_LINKS=https://github.com/ddelange/vaex/releases/expanded_assets/core-v4.17.1.post4
pip install --force-reinstall vaex

fixes #2366, fixes #2368, fixes #2397

maartenbreddels · 2023-01-21T08:32:29Z

Hoi 👋

exciting, will take a look early next week!

manylinux takes around 2.5hrs per wheel and alpine arm64 up to 4 hrs

that worries me a bit.. :)

groeten,

Maarten

ddelange · 2023-01-21T09:40:18Z

here are all timings: https://github.com/ddelange/vaex/actions/runs/3965720337/usage

depending on how often a month you release vaex, this could eat into the 2k free minutes of GH...

as the parallelization is maximised and they're pushed to PyPI as soon as they're built, most of the wheels will be available soon upon release regardless

here are all the wheels: distributions.zip

ddelange · 2023-01-21T10:32:44Z

interestingly, that was 8260 minutes ^

apparently that's OK? then I don't understand their explanation 🤔 https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#included-storage-and-minutes

ddelange · 2023-01-21T10:40:57Z

ah there is a fair amount of duplication in that usage table for whatever reason 🤯

ddelange · 2023-01-22T10:54:32Z

a diff of current PyPI vs the zip above:

 vaex_core-4.16.1-cp310-cp310-macosx_10_9_x86_64.whl
 vaex_core-4.16.1-cp310-cp310-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp310-cp310-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp310-cp310-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp310-cp310-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp310-cp310-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp310-cp310-win_amd64.whl
+vaex_core-4.16.1-cp311-cp311-macosx_10_9_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-macosx_11_0_arm64.whl
+vaex_core-4.16.1-cp311-cp311-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp311-cp311-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-musllinux_1_1_aarch64.whl
+vaex_core-4.16.1-cp311-cp311-musllinux_1_1_x86_64.whl
+vaex_core-4.16.1-cp311-cp311-win_amd64.whl
 vaex_core-4.16.1-cp36-cp36m-macosx_10_9_x86_64.whl
-vaex_core-4.16.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp36-cp36m-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp36-cp36m-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp36-cp36m-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp36-cp36m-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp36-cp36m-win_amd64.whl
 vaex_core-4.16.1-cp37-cp37m-macosx_10_9_x86_64.whl
-vaex_core-4.16.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp37-cp37m-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp37-cp37m-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp37-cp37m-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp37-cp37m-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp37-cp37m-win_amd64.whl
 vaex_core-4.16.1-cp38-cp38-macosx_10_9_x86_64.whl
 vaex_core-4.16.1-cp38-cp38-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp38-cp38-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp38-cp38-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp38-cp38-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp38-cp38-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp38-cp38-win_amd64.whl
 vaex_core-4.16.1-cp39-cp39-macosx_10_9_x86_64.whl
 vaex_core-4.16.1-cp39-cp39-macosx_11_0_arm64.whl
-vaex_core-4.16.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+vaex_core-4.16.1-cp39-cp39-manylinux_2_28_aarch64.whl
+vaex_core-4.16.1-cp39-cp39-manylinux_2_28_x86_64.whl
+vaex_core-4.16.1-cp39-cp39-musllinux_1_1_aarch64.whl
 vaex_core-4.16.1-cp39-cp39-musllinux_1_1_x86_64.whl
 vaex_core-4.16.1-cp39-cp39-win_amd64.whl

ddelange · 2023-01-23T07:30:45Z

packages/vaex-core/src/hash_object.cpp

-namespace std {
-    template<>
-    struct hash<PyObject*> {
-        size_t operator()(const PyObject *const &o) const {
-            return PyObject_Hash((PyObject*)o);
-        }
-    };
-}


This was my best guess...

pybind11 ref https://github.com/pybind/pybind11/blob/769fd3b889fef6cddb060f2a0be26aee62b4da05/include/pybind11/pytypes.h#L859

failure log https://github.com/ddelange/vaex/actions/runs/3965609112/jobs/6795506653#step:6:2110

@maartenbreddels any thoughts on this (incl me updating the pybind11 submodule)?

ddelange · 2023-01-23T07:31:30Z

packages/vaex-core/setup.py

@@ -183,12 +183,14 @@ def __str__(self):
      include_package_data=True,
      ext_modules=([extension_vaexfast] if on_rtd else [extension_vaexfast, extension_strings, extension_superutils, extension_superagg]) if not use_skbuild else [],
      zip_safe=False,
+      python_requires=">=3.6",


cibuildwheel parses this to determine which wheels to build

cc @franz101

see also the diff above

ddelange · 2023-02-27T07:19:30Z

I'm guessing this is blocked by #2339

maartenbreddels · 2023-03-06T13:11:53Z

Just letting you know i'm very busy and had a vacation.
Yes, I'll try to get #2339 green first!

ddelange · 2023-05-25T23:20:01Z

fwiw there are now third party free minutes on native arm64 machines, to get rid of the slow qemu builds

maartenbreddels · 2023-07-10T20:25:06Z

Could you try rebasing this?

ddelange · 2023-07-10T20:48:15Z

@maartenbreddels already merged in master 👍

ddelange · 2023-07-11T06:55:46Z

    ERROR: Could not find a version that satisfies the requirement vaex-core<4.17,>=4.17.0 (from vaex)
    ERROR: No matching distribution found for vaex-core<4.17,>=4.17.0

maartenbreddels · 2023-07-11T15:50:10Z

Yeah, a bug/artifact or our release script. Should be good now.

ddelange · 2023-08-03T20:50:04Z

hoi @maartenbreddels 👋

I pulled master and fixed merge conflicts, but it looks like CI is still not very happy. Seeing errors like hdf file missing on disk, and TypeError: train() got an unexpected keyword argument 'early_stopping_rounds'.

Do you think it might be related to this PR?

franz101 · 2023-08-17T20:51:39Z

Just wondering here on the Python packaging. Python 3.6 and 3.7 are now deprecated on the other hand we can bump to 3.10 and 3.11?

to-bee · 2023-08-28T07:48:57Z

Do we have any updates on this MR?

ddelange · 2023-09-01T08:07:30Z

HI @maartenbreddels 👋

Was your s3 account deleted by any chance?

vaex.open('s3://vaex/taxi/yellow_taxi_2009_2015_f32.hdf5?anon=true')

raises

FileNotFoundError: [Errno 2] Path does not exist 'vaex/taxi/yellow_taxi_2009_2015_f32.hdf5'. Detail: [errno 2] No such file or directory

maartenbreddels · 2024-07-04T10:06:12Z

readthedocs is green again and looks good

😍

ddelange · 2024-07-04T11:04:46Z

@maartenbreddels can you cancel some of the older Actions runs here, to free a slot for the latest commit?

maartenbreddels · 2024-07-04T13:09:25Z

Maybe include

concurrency:
  group: ${{ github.ref }}
  cancel-in-progress: true

like in pythonpackage.yaml ?

ddelange · 2024-07-04T14:06:25Z

yep! added a slightly more sophisticated version

setu4993 · 2024-07-04T14:51:29Z

Exciting to see all those ✅s!

ddelange · 2024-07-04T23:02:43Z

ci/conda-env.yml

@@ -54,3 +54,5 @@ dependencies:
 - python-utils
 - progressbar2
 - zipp<3.16.0
+- pip:
+  - lightgbm>=4.0.0


re: flaky micromamba

I could replace setup-micromamba with setup-python:

you often see packages go pip install pandas[dev]. we could move this stuff into vaex[dev] and deprecate this file? and github actions goes pip install -e .[dev]?

on a side-note: shouldn't the lightgbm>=4.0.0 constraint go into vaex-ml pyproject.toml anyway?

This file was there to quickly and reliably create a conda environment. This previously was impossible to do with pip.
I'm afraid that trying this will take a lot of time. Maybe we can merge this, do a release, and try to do this separately in a new PR?

maartenbreddels · 2024-07-05T07:56:53Z

.github/workflows/wheel.yml

+    - name: Upload release assets
+      if: github.event_name == 'release'
+      uses: softprops/[email protected]
+      with:
+        files: packages/vaex-core/dist/*


We are not using the release feature of github, but release on tags, looking at the docs, it seems we should have a similar if as 'Publish a Python distribution to PyPI' right?

this step will upload wheels to a github release, eg https://github.com/ddelange/python-magic/releases/tag/0.4.28.post7

it is a no-op if there is no release

maartenbreddels · 2024-07-05T08:00:38Z

One thing i'd like to see, but I'm not sure where this would go. Is to try to do a simple import vaex; vaex.example() for the build wheels, so we know the wheels are valid.
Not sure if this is easy to do, I'm just afraid that after this many changes we might see a broken release.

@ddelange I believe you've been testing with this branch, so if you are confident that is not the case, let me know and we can skip this step.

I think we should also expand the test matrix, because although we make the wheels for 3.12, we did not run the test suite.

.github/workflows/wheel.yml

ddelange · 2024-07-05T13:16:10Z

were the musllinux wheels ever tested on an alpine box? musllinux smoketest is currently failing due to a missing pyarrow wheel: apache/arrow#40177

see also last three commits. i would suggest either skipping these platform smoketests, or skipping musllinux wheels. building pyarrow from source is a bit of a pain i think. any thoughts?

maartenbreddels · 2024-07-05T13:19:07Z

let's skip it (but maybe add comments on how to enable it when possible in the future?)

ddelange mentioned this pull request Jan 21, 2023

Maximize wheel build parallellization ddelange/vaex#1

Closed

ddelange commented Jan 23, 2023

View reviewed changes

ddelange force-pushed the build-matrix branch from beff33a to 7bf26cc Compare February 27, 2023 07:15

This was referenced Jun 30, 2023

[FEATURE-REQUEST] Support Python 3.11.2 #2368

Open

[BUG-REPORT] Failed to build vaex-core (pybind11 issue?) #2366

Open

ddelange changed the title ~~Build aarch64 wheels~~ Build aarch64 wheels and support python 3.11 Jul 10, 2023

maartenbreddels force-pushed the master branch from 8d44942 to d4bf017 Compare July 11, 2023 13:55

CarrieFilion mentioned this pull request Aug 2, 2023

Install with Python 3.11 fails athob/py-ananke#1

Closed

ddelange referenced this pull request in rapidfuzz/RapidFuzz Aug 10, 2023

upgrade cython

dfea552

ddelange mentioned this pull request Aug 30, 2023

Discussion: support bundling libmagic ahupp/python-magic#233

Closed

ddelange force-pushed the build-matrix branch 3 times, most recently from 5680eb9 to 2136629 Compare September 4, 2023 08:28

ddelange added 3 commits July 4, 2024 13:27

Remove autocancel, doesn't work from forks

eacef43

Add cancel-in-progress keys

3fdd98e

Amend to only cancel on PRs

8fb9b3e

ddelange commented Jul 4, 2024

View reviewed changes

maartenbreddels reviewed Jul 5, 2024

View reviewed changes

ddelange commented Jul 5, 2024

View reviewed changes

.github/workflows/wheel.yml Show resolved Hide resolved

ddelange added 5 commits July 5, 2024 12:20

Add CIBW_TEST_COMMAND

7a78a23

Disallow numpy v2

1b58e08

Skip musllinux due to misding blake3 wheels

4fc584d

Install carco for blake3 source install

3536f89

Fix CIBW_ENVIRONMENT

808a0de

ddelange and others added 11 commits July 5, 2024 16:26

Add CIBW_TEST_SKIP

169a236

Build wheels with numpy v1

735c319

run tests on 3.10-3.12

f371ba7

upgrade micromamba

777b0bf

unpin pytest-asyncio

5b9d775

change numpy pin

c68d2b5

unpin uvicorn

eb23c9e

use object instead of np.object to support numpy

c7ab3dd

use float64 instead of np.float to support numpy >1.20

080a747

unpin xarray for python 3.12

b25a9e7

try install lightgbm with conda, not pip

576ebfb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support CPython 3.11, 3.12, and aarch64 processors #2331

Support CPython 3.11, 3.12, and aarch64 processors #2331

ddelange commented Jan 20, 2023 •

edited

Loading

maartenbreddels commented Jan 21, 2023

ddelange commented Jan 21, 2023 •

edited

Loading

ddelange commented Jan 21, 2023

ddelange commented Jan 21, 2023 •

edited

Loading

ddelange commented Jan 22, 2023

ddelange Jan 23, 2023 •

edited

Loading

ddelange Feb 27, 2023

ddelange Jan 23, 2023

ddelange Aug 17, 2023

ddelange commented Feb 27, 2023

maartenbreddels commented Mar 6, 2023

ddelange commented May 25, 2023

maartenbreddels commented Jul 10, 2023

ddelange commented Jul 10, 2023

ddelange commented Jul 11, 2023

maartenbreddels commented Jul 11, 2023

ddelange commented Aug 3, 2023

franz101 commented Aug 17, 2023

to-bee commented Aug 28, 2023

ddelange commented Sep 1, 2023 •

edited

Loading

maartenbreddels commented Jul 4, 2024

ddelange commented Jul 4, 2024 •

edited

Loading

maartenbreddels commented Jul 4, 2024

ddelange commented Jul 4, 2024

setu4993 commented Jul 4, 2024

ddelange Jul 4, 2024 •

edited

Loading

maartenbreddels Jul 5, 2024

maartenbreddels Jul 5, 2024

ddelange Jul 5, 2024

maartenbreddels commented Jul 5, 2024

ddelange commented Jul 5, 2024

maartenbreddels commented Jul 5, 2024

Support CPython 3.11, 3.12, and aarch64 processors #2331

Are you sure you want to change the base?

Support CPython 3.11, 3.12, and aarch64 processors #2331

Conversation

ddelange commented Jan 20, 2023 • edited Loading

the wheels from this PR can be installed with:

maartenbreddels commented Jan 21, 2023

ddelange commented Jan 21, 2023 • edited Loading

ddelange commented Jan 21, 2023

ddelange commented Jan 21, 2023 • edited Loading

ddelange commented Jan 22, 2023

ddelange Jan 23, 2023 • edited Loading

Choose a reason for hiding this comment

ddelange Feb 27, 2023

Choose a reason for hiding this comment

ddelange Jan 23, 2023

Choose a reason for hiding this comment

ddelange Aug 17, 2023

Choose a reason for hiding this comment

ddelange commented Feb 27, 2023

maartenbreddels commented Mar 6, 2023

ddelange commented May 25, 2023

maartenbreddels commented Jul 10, 2023

ddelange commented Jul 10, 2023

ddelange commented Jul 11, 2023

maartenbreddels commented Jul 11, 2023

ddelange commented Aug 3, 2023

franz101 commented Aug 17, 2023

to-bee commented Aug 28, 2023

ddelange commented Sep 1, 2023 • edited Loading

maartenbreddels commented Jul 4, 2024

ddelange commented Jul 4, 2024 • edited Loading

maartenbreddels commented Jul 4, 2024

ddelange commented Jul 4, 2024

setu4993 commented Jul 4, 2024

ddelange Jul 4, 2024 • edited Loading

Choose a reason for hiding this comment

maartenbreddels Jul 5, 2024

Choose a reason for hiding this comment

maartenbreddels Jul 5, 2024

Choose a reason for hiding this comment

ddelange Jul 5, 2024

Choose a reason for hiding this comment

maartenbreddels commented Jul 5, 2024

ddelange commented Jul 5, 2024

maartenbreddels commented Jul 5, 2024

ddelange commented Jan 20, 2023 •

edited

Loading

ddelange commented Jan 21, 2023 •

edited

Loading

ddelange commented Jan 21, 2023 •

edited

Loading

ddelange Jan 23, 2023 •

edited

Loading

ddelange commented Sep 1, 2023 •

edited

Loading

ddelange commented Jul 4, 2024 •

edited

Loading

ddelange Jul 4, 2024 •

edited

Loading